Cross-Validation Optimization for Structured Hessian Kernel Methods

We propose a highly efficient framework for kernel multi-class models with a largeand structured set of classes, and more general for penalized likelihood kernel methods.As opposed to many previous approaches which try to decompose the fitting probleminto many smaller ones, we focus on a Newton optimization of the complete model,making use of model structure and linear conjugate gradients in order to approxi-mate Newton directions. Crucially, our learning method is based entirely on matrix-vector multiplication primitives with the kernel matrices and their derivatives, allowingstraightforward specialization to new kernels, and directing the focus of code optimiza-tion to these primitives.Kernel parameters are learned automatically by maximizing the cross-validation loglikelihood, and predictive probabilities are estimated. We demonstrate our approach onlarge scale text classification tasks with hierarchical structure on throusands of classes,achieving state-of-the-art results in an order of magnitude less time than previous work.We also discuss an extension to label sequence learning with kernel conditional randomfields.